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class TObjectlnvariant : public TDatalnvariant 
public: 

virtual Boolean Test (const TDatum&); 
virtual Boolean IsValidNonNilOjbjectPointer 

(const, void*) const; 

class TCollectiblelnvariant : public TDatalnvariant 
public: 300 
virtual Boolean Test (const TDatum&); / 
virtual Boolean IsEqualToItself (const MCollectible&) 

const; 

virtual Boolean DoStreamingCheck(const 

McCollectible&) const; 

class TCollectionlnvariant : public TDatalnvariant 
public: 310 
virtual Boolean Test(const TDatum&); / 
virtual Bong GetlterateCount (const TCollection& ) 

const; 

virtual Boolean DoContentsInvariancyCheck (const 

TCollection&) const. 



FIGURE 3 
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Boolean TObjectInvariant::Test (const TDatum& datum 
(const void* object = (void*) datum.GetData (); 
if (object == NIL) 400 

return FALSE;/ / pointer is NIL 
if (IsValidNonNilObject Pointer (object) ) 

return FALSE;// non-zero pointer isn't valid 
return TRUE;// we passed all our intra-invariant checks; 

Boolean TCollectibleInvariant::IsEqualToItself (const 

McCollectible& object) const (return collectibleisEqual 
(object) //is object equal to itself? 

Boolean TCollectibleInvariant::Test (const TDatum& datum) ( 
const McCollectible* object = (McCollectible*) 
datum.Get Data (); 
if (!IsEqualToItself(*object)) 

return FALSE;// object not equal to itself 
if (collectible- >GetClass Name() == NIL ) 

return FALSE;/ /object lacks class name 
if ( !DoContentsInvariancyCheck(*object)) 

return FALSE;/ /found a streaming problem 
return TRUE; / / we passsed all our intra-invariant 
checks; 

longTColectionInvariant::GetIterateCount 

(const TCollection& c) (TIterator* i = c.CreateIterator(); 

McCollectible*m;long n=0; 410 

for (m=i>First(); m!=NIL; m=i<Next()0 n++/ /count 

members iterated over 

delete i; return n; / / return the iteration count 

Boolean TCollectionInvariant::Test (const TDatumfc datum) ( 
const Tcoliection* C= (TCollection*) dataum.GetData (); 
if ( GetIteratorCount(*c) !=C-.Count() ) 



FIGURE 4 
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class TCollectionAddPrecondition :public TFunctionCondition ( 
public: 500 
virtual Boolean Test (TFunctionData&); 

I 

Boolean TCollectionAddPrecondition::Test 

(TFunctionData& data) 
const TCollection* c = TCollection*) data.GetThis ); 
const MCollectible* )McCollectible*) 

data.GetParameter(l); 510 
const McCollectible* f = ITerator(*collection). First () 
if ((f!=NIL) && (stromp (o->GetClassName()>> 

GetClassName() 0 =0)) 
return FALSE: / /homogeneity violation 

(class names digger) 
return True / /we can add the collectible to the collection 



FIGURE 5 
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class TCollectionAddPostcondition : public 

TFunctionCondition 

public: 600 

virtual Boolean Test(TFunctionData&); 



Boolean TCollectionAddPostcondition::Test( TFunctionData& 

data ) ( 

const TCollection* c = (TCollection*) data.GetThis (); 
const McCollectible* o = (McCollectible*) 

data.GetParameter(l); 
const McCollectible* f = TIterator(*collection).First(); 

610 

if (c->Member(*) == FALSE) 

return FALSE; / /collectible is not in the collection 

long postAddCount = c->Count(); 
Boolean countDecreased = (postAddCount< gPreCount); 
Boolean countlncreasedMoreThanl = (postAddCount > 
(gPreCount + 1)); 

if (IgWasMember && postAddCount!=(gPreCount+l)) 
return FALSE; / /count should have incremented 

if (gWasMember && 

(countDecreased I I countlncreasedMoreThanl)) 
return FALSE; / / count should not have decreased or 

increased more than 1 
return TRUE; //all our assertions passed. 



FIGURE 6 
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METHOD AND APPARATUS FOR 
GENERATING TEST DATA FOR AN 
AUTOMATED SOFTWARE TESTING 
SYSTEM 

This application is a continuation of allowed prior appli- 
cation Ser. No. 08/3 10,455 filed Sep. 22, 1994 issued on Sep. 
27, 1995 as U.S. Pat No. 5.490,249 which is a continuation 
of parent application Ser. No. 07/996.047 filed Dec. 23. 1992 
issued Feb. 14, 1995 as U.S. Pat No. 5,390325. 

FIELD OF THE INVENTION 

This invention generally relates to improvements in test 
systems for testing software applications and more particu- 
larly to automating the test process for testing software 
applications. 

BACKGROUND OF THE INVENTION 

Among developers of workstation software, one of the 
most pressing requirements is for a software reliability 
program which can automatically test each of the compo- 
nents of a software application and identify problems, some- 
times referred to as "bugs", in the application, and report the 
problems. To date, this processing is tedious and performed 
by a team of persons who identify test sequences and 
perform the tests on the software application. 

Some attempts have been made at interactively debugging 
programs by setting break points and other traps in an 
application and observing the application as it executes. This 
approach was patented by DEC in U.S. Pat. No. 4,589,068. 
However, this approach could not test the unique character- 
istics of each subroutine/function of a software application 
and resulted in incomplete test operations. 

SUMMARY OF THE INVENTION 

Accordingly, it is a primary objective of the present 
invention to automate the testing of a software application. 
A computer system is employed to input design specifica- 
tions for each of the subroutines that make up a software 
application that is to be the target of a test and the express 
preconditions and postconditions based on die design speci- 
fications. Internal databases are constructed to maintain 
target type relationships, subroutine declaration syntax and 
data requirements, condition-subroutine associations, and 
invariant-type associations. 

The preconditions and postconditions are compiled and 
link edited into a shared library and mapped to the particular 
subroutine. Then, once the databases have been loaded, 
wrappers are created to generate the source code for func- 
tions employed by the invention to perform actual target 
subroutine calls. The source code is written to a text file on 
a disk. The wrapper source code must be compiled and link 
edited into the shared library containing the precondition, 
postcondition and invariant subroutines. 

Daring testing, the invention loads parameter data for 
each subroutine selected for test Then, for all data instances 
defined for each of the subroutines to be tested, an invariant 
function is performed and the resulting data is checked to 
ascertain its validity. If the data is invalid, then it is reported. 
However, if the data is not invalid, then the next data 
instance is performed until all data instances have been 
performed for each of the subroutines. 

Each subroutine is next called with a precondition, if the 
precondition is true, then the subroutine is called with data 
from each of the predefined data instances. For each of the 
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instances, if the data is valid, then the next instance is 
invoked. However, if the data is invalid, then the invalid data 
is reported. The same processing is performed for each of the 
subroutines on the postconditions and similar exception 
5 processing is reported. In this manner, every subroutine of 
an application can be automatically tested and exceptions 
noted for any instance, precondition and postcondition. 

BRIEF DESCRIPTION OF THE DRAWINGS 

10 FIG. 1 is a block diagram of a personal computer system 

in accordance with the subject invention; 
FIG. 2 is a hierarchical drawing of the subject inventions 

processing hierarchy in accordance with the subject inven- 
15 ti 011 ; 

FIG. 3 is an invariant class declaration for generic objects 
in accordance with the subject invention; 

FIG. 4 is a nontrivial invariant method in accordance with 
the subject invention; 
20 FIG. 5 illustrates an example precondition in accordance 
with the subject invention; 

FIG. 6 is a postcondition class in accordance with the 
subject invention; and 

25 FIG. 7 is a block diagram of major steps in accordance 
with the subject invention. 

DETAILED DESCRIPTION OF THE 
INVENTION 

30 The invention is preferably practiced in the context of an 
operating system resident on a personal computer such as 
the IBM PS/2 or Apple Macintosh computer. A representa- 
tive hardware environment is depicted in FIG. 1, which 
illustrates a typical hardware configuration of a workstation 

35 in accordance with the subject invention having a central 
processing unit 10, such as a conventional microprocessor, 
and a number of other units interconnected via a system bus 
12. The workstation shown in FIG. 1 includes a Random 
Access Memory (RAM) 14, Read Only Memory (ROM) 16, 

40 an I/O adapter 18 for connecting peripheral devices such as 
disk units 20 to the bus, a user interface adapter 22 for 
connecting a keyboard 24, a mouse 26, a speaker 28, a 
microphone 32, and/or other user interface devices such as 
a touch screen device (not shown) to the bus, a communi- 

45 cation adapter 34 for connecting the workstation to a data 
processing network and a display adapter 36 for connecting 
the bus to a display device 38. The workstation has resident 
thereon an operating system such as the IBM OS/2 operating 
system or the Apple OPUS operating system. 

50 

Issues 

There are a number of unresolved issues in prior art 
software reliability testing which must be addressed to 
55 improve the quality strategy and testing systems. The fol- 
lowing issues are addressed by the subject invention and a 
preferred embodiment of the invention solving each of the 
issue is presented below. 

Reliability 

The term reliability has many definitions. However, for 
purposes of the subject invention, the following definition is 
incorporated, "the probability of failure-free operation of a 
computer program for a specified time in a specified envi- 
65 ronment" Software Reliability represents the proven ability 
of an implementation to agree with its specification. Reli- 
ability must be subject to objective measurements — hence 
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use of the word probability — based upon repeated execu- suffice. However, software is much marc complex than a 

tions of software under real-world operating conditions hard disk drive. Measuring software MTBF requires gen- 

which satisfy design specifications. This implies that reli- eration of an operational profile which defines the function 

ability of any software component can and must be objec- calls made during the reliability test interval 

tively represented using measures of software as they are 3 There is a problematic reason why software MTBF raea- 

presented below. sures m virtually nonexistent: nobody has yet developed a 

Software reliability depends on a failure-free, computer universal scheme far deteimining function call sequences 
program, at a specified time, and in a specified environment during MTBF analysis. Calling the same unit tests repeat- 
It is necessary that all our definitions be resolvable down to edly doesn't accurately represent the system's complex 
concrete definitions. The following are definitions of those io combmatcrial possibilities. Replaying Graphical User Inter- 
terms within the context of measuring the reliability of an face (GUI) event sequences usually activates only a small 
application program's interface (AH): portion of the total system code. Besides, GUI actions are 

failure-free — observed operation does not disagree with usually not portable between dissimilar operating systems 

AH specification. and cannot offer fair comparisons. 

computer program — API implementation under test 15 Operational profile generation for reliability analysis is a 

specified time — time spent within explicit API function challenging problem, but not an impossible one. For 

calls during test example, here are several steps which define an operational 

specified environment — states and actions which do not profile: 

disagree with API specification. 2 o ^ or m Y va ^ state, determine the entire domain of 

Note the deliberate use of "do not disagree" rather than subroutines and parameter data which can be executed next 

"do agree**. This is because any reasonable action which is according to a design specification; 

not explicitly disallowed by a specification should be 2. choose one subroutine and corresponding parameter 

allowed. This forces design engineers and technical writers data from that domain; 

to provide thorough operating specifications for every public 2 5 3. execute that subroutine with its chosen parameter data; 

interface for test engineers, automated testing programs, and 4> perform validity cnecks to determine if a failure arose 

client developers. during the call; and 

Measures 5. if no failure arose, then perform step 1), else report the 

Reliability is the probability that a product will operate 30 failure 811(1 st0 P- 

correctly for a specified time. This statement conversely A variant of this scheme is commonly referred to as a 

implies that the time a product will probably operate cor- random walk through. Any system which can perform 

rcctly is also a measure of reliability. This time measure is random walk through with an API will have an inherent 

often expressed as mean time between failure (MTBF) capability to perform reliability measurement. The auto- 

which represents the average duration before a product 35 mated testing system of the subject invention can perform 

breaks down. MTBF measures help purchasers select a random walk through far any multi-component API of 

product offering a superior operating life from others having arbitrary size, thus providing the inherent capability to 

similar features. Manufacturers can also use MTBF metrics perform MTBF measurement for most components, 

to objectively determine which components in critical sys- The system proposed in this document offers a techno- 

terns require priority attention for overall system reliability ^ logical breakthrough to the entire software industry because 

improvement it is possible to compare the reliability of competitive but 

Nearly every computer hard disk drive has an MTBF diverse products such as Windows NT, Macintosh, and 

rating, but such ratings are uncommon for software. This OS/2. The ability to perform random walk through offers 

situation is peculiar because the same techniques which another breakthrough in software testing — the statistical 

provide MTBF ratings for hardware can also be directly 45 quality control technique of random sampling can now be 

applied to software, but virtually no software manufacturers applied to software, 
employ MTBF. The subject invention provides the report 

information to generate an accurate computation of MTBF. Testing 

The first personal computer software manufacturer to Testing has historically included a range of activities from 

offer reliability ratings for its products would be outstanding 50 writing test cases to improving product designs. However, in 

in the industry for several reasons. First, the recognition far many cases the exact duties of test engineers has been 

being the first to offer such measures has substantial merit by unclear. Unclear objectives often results in unachieved 

itself. Second, the ability to measure software reliability objectives. Therefore, the following objectives were devel- 

proves that the company has a thorough and controllable oped to fashion a solution to the shortfalls of current 

software testing process. 55 software testing technology. / 

Software reliability ratings are uncommon for speculative Test engineers should focus oh assuring that their 

reasons. One reason is software is not a physical item like a assigned components operate according to design specifica- 

hard disk drive; it cannot be picked up and measured with a tions to ensure errors are measured, controlled, and 

scale. Another possible reason is that statistical quality improved throughout the product life cycle. Breaking new 

control — the discipline which uses MTBF ratings — is typi- 60 ground in software quality requires that test engineers 

cally taught to mechanical and electrical engineers, but not include software reliability measurement in their problem 

to software engineers or computer scientists. Software domain. The traditional testing tasks can still be performed, 

developers are not instructed in the importance of statistical but measuring and controlling product reliability should be 

quality control, consequently the software development pro- recognized as their highest priority task, 

fession has not widely applied it 65 Software testing involves detecting differences between 

Measuring the reliability of hard disk drives is not diffi- design specification and implementation operation. This 

cult; randomly reading and writing until a failure occurs will activity raises several qu estions: 
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How should software design be specified? 

How should implementation be verified? 

How well does implementation meet specification? 

There are many models for representing how software 
should work. Some models are more appropriate than others 
depending on specification purpose. Since software testing 
involves evaluating the outcome of experimental trials, our 
specification model should inherently support decision 
evaluation. There are three important decisions in software 
testing: 

Is state X valid? 

Should future action Y work? 

Did past action Z work? 

Any software component can be adequately and automati- 
cally tested by consistently applying three types of criteria to 
software with an automated testing system. The subject 
invention offers concrete implementations to implement 
these three criteria. 

Despite the relative maturity of the software industry and 
the investment of many dollars, fundamental questions of 
software testing remain unanswered, for example: 

What are the test cases for function X?Functions XI. XS, 
. . . XN? 

What are the test cases for structure Y?Structures Yl, Y2, 
. . . YN? 

Has component Z been tested thoroughly?Components 
Zh Z2, . . . ZN? 

Incredibly, these basic issues are still unresolved, or even 
worse, haphazardly addressed by test engineers in a random 
manner. Which explains why software is notorious for 
reliability problems — no two test engineers will test identi- 
cal products in an identical manner and produce repeatable 
results. A globally unrepeatable process will also be globally 
inconsistent and uncontrollable. 

Testing an API associated with an operating system 
demands a consistent process to verify correct operation of 
all methods and classes. For any API there should be a 
systematic way to define a core set of test cases which must 
successfully run to verify correct operation of the interface. 

An ideal testing strategy should mimmize human effort 
while maximizing the number of latent defects found in new 
or modified product code. It should also be designed to 
answer the basic problems which all test engineers face. 

Measures 

There are several measures for the functionality of sys- 
tems which perform software testing. One measure, "effort" 
represents how many resources are typically required to find 
a defect in a product, and can be expressed in dollars or 
person-hours. Another measure called effectiveness repre- 
sents the system's overall ability to find new bugs, and can 
be expressed in bugs per week. Another measure called 
efficiency — the ratio of effectiveness divided by effort — 
represents how economically defects are found, and can be 
expressed in dollars or hours spent finding a bug. 

Efficiency measures are extremely important because they 
can be used for objective selection of the best among 
multiple systems for testing a specific product Given many 
systems, the one with the highest efficiency rating is the best 
investment. Efficiency measures can also be used to deter- 
mine when a product is ready for release; as testing effi- 
ciency drops — when more testing effort reveals fewer new 
defects — the product becomes demonstrably reliable. 

Efficiency ratings mathematically depend on both effort 
and effectiveness ratings. Fortunately, effectiveness ratings 
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are usually automatic byproducts of testing administration 
and can be constructed from bug database queries. Effort 
ratings are not so easily obtained. They are essential, but also 
unstandardized. Test person-hours, test cases run, or test 

5 CPU time are all plausible measures, but it is not always 
obvious which should be used. Useful metrics such as 
person-hours are not — and will probably never — be main- 
tained by administrative or testing systems especially when 
test engineers are salaried or do not keep hourly work 

10 records. 

Any testing system must automatically define and record 
testing effort because accurate evaluation of system capa- 
bility and product reliability cannot occur without it The 
subject invention proposes a revolutionary technique for 
15 measuring test effort 

DETAILS OF THE SUBJECT INVENTION 

This section addresses issues within the context of an 
ideal testing system. An ideal testing system should have the 
20 following characteristics: 

offer superior testing efficiency (effect effort); 
perform thorough unit and integration testing; 
exploit existing target syntax (headers, source, binaries); 
25 aD ^ 

report how, why, and where errors arose. 
An ideal system must focus human effort on writing 
reusable descriptions of how software should work rather 
than generating specific test cases where software should or 
30 should not work correctly. Experience suggests that software 
operation can be usually described with a finite amount of 
human effort whereas the theoretical number of valid and 
unique test cases cannot 
The following section explain the detailed implementa- 
35 tion of the subject invention. The following are major 
techniques used by the subject invention: 

1. generate a test domain of type-legal operations using 
syntactic type declarations; 

2. use preconditions to prevent type-legal but design- 
40 illegal operations from that domain; 

3. use selectors to choose a subroutine and data from that 
domain for execution; 

4. use postconditions to detect invalid operations after 
each explicit subroutine call; 

45 5. use invariants to detect invalid data states after each 
explicit subroutine call; 

6. use stores to maintain data for reuse across explicit 
subroutine calls. 
The following sections present components in detail and 

50 illustrate how these components combine to address many of 
the outstanding testing issues mentioned earlier. Precondi- 
tions and postconditions provide the key functionality for 
automated testing because they can be used to represent the 
specifications and verify the implementation of the API 

55 under test Test engineers write preconditions and postcon- 
ditions rather than individual test cases. 

Test Domain Generation 

In a generic sense, a test domain is a sequence of explicit 
60 subroutine calls and environmental conditions which con- 
stitute valid test cases for a target API Test domains are 
generated by analyzing data type references declared in an 
API design specification. Data type references and data 
instance availability arc employed to determine which sub- 
65 routines and data are candidates for execution based upon 
the data types required as subroutine parameters and the 
actual data instances that are available. 
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Far example, assume that TPerson and TName instances 
exist along with a subroutine TPerson: :SetName(TName). 
With this information, a determination is made mat SctName 
is a candidate for execution because instances are available 
which satisfy parameter type requirements. 

lest domains generated using only C++ data types contain 
invalid test cases because data values are not considered 
when selecting candidate subroutines. For example, the 
specification for SctName might permit only certain TName 
values such as "John" but not 4, 1@#$%". This example 
illustrates why type requirements by themselves cannot 
identify valid test domains. In addition to syntactical type 
rules, semantic rules are also required to express operation 
requirements that cannot be derived from syntax declara- 
tions alone. 

The subject invention is a framework for testing program- 
matic interfaces by systematically exploring valid call 
sequences defined by a collection of subroutines with a 
collection of data. The subject invention does not explicitly 
write unit tests. Instead, it provides the tools to develop rules 
which model and verify the correct operation of software 
while leaving the complexities of test selection, execution, 
sequencing, coverage, and verification to an automated 
system. The number of test scenarios generated by the 
system depends upon cumulative execution time and is 
reliably independent of human effort. 

There exist possibly new, obscure, or overworked terms in 
the context of this invention. To improve understanding and 
avoid project, brief descriptions of frequently used abstract 
terms and concepts are provided below. 

syntactic protocol requirements of operation in terms of 
data type relationships. 

semantic protocol requirements of operation beyond data 
type relationships. 

predicate mechanism for evaluating validity and produc- 
ing a true or false logical result; two types are invariants 
which validate states, and conditions which validate actions. 

invariant mechanism for validating an observable state; 
invariants should minimally impact the state being observed. 

condition mechanism for validating an action; two sub- 
types are preconditions which evaluate if future actions 
should operate correctly, and postconditions which evaluate 
if past actions did operate correctly. 

precondition mechanism which determines if an action 
can be initiated correctly given the observable state imme- 
diately before the action. 

postcondition mechanism which determines if an action 
completed correctly given the observable state immediately 
after the action. 

space conceptual or physical system having one or more 
degrees of freedom containing closely related elements; 
elements within a space are generally countable, iteratable, 
and finite. 

type space hierarchical system of data types, usually a 
directed acyclic graph. 

function space system of typed subroutines organized by 
a type space. 

data space system of typed data organized by a type space, 
predicate space system of typed predicates organized by 
a type space. 

test space unification of type, function, and data, spaces 
into a complete entity. 

target focus of a test; classes, subroutines, and objects can 
all be targets. 
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step smallest topological transition in a space, 
exploration one or mare steps through a space; a random 
walk is an exploration. 

function computer subroutine; two subtypes are method 
which requires the context of an object and procedure which 
does not require such a context 

method non-static member function; virtual functions are 
always methods. 
io procedure global subroutine or static member function. 

defect unit of discrepancy between how software should 
operate and how it actually does; a bug that has not yet been 
found; a defect can cause many failures or none. 

failure observable effect of one or more defects a 'bug' 
15 that has been found; unintended function results, outcomes, 
or effects can be failures; a 'crash* is almost always a failure. 

Test scenarios are automatically generated by the system 
using many information types representing characteristics 
and protocol of the code under test. The following sections 
20 present the information required by the system to perform a 
test 

A primary information type is syntactic protocol which 
describes the formal interface of the target code from a 
low-level, lexical perspective. The system analyzes target 

25 code syntax to learn type relationships, class methods, and 
data parameters. C++ declarations are the primary source of 
syntactic protocol. Minimal effort is required by a test 
engineer to provide syntactic protocol since it is readily 
ascertainable from target source files and is analyzed auto- 

30 matically by the system's lexical parser. 

The following syntactic protocol is employed by the 
subject invention: 



class MColIectible{ public: 

virtual Boolean IsEqual (coast MCollectible&) 

canst 

virtual const char GctClassName( ) const; 

}; 

class TCollection: public MCollectible 
40 public: 

virtual MCollectible* Add (MCollectible*) = O; 
virtual MCollectible RemDve(const MCollcctible&) 
= 0; 

virtual long count () const = 0; 
virtual TItcrator* CrcatcDeratar ( ) const = 0; 
45 virtual BookanlsEqual ( ) const MCollcctiblc&) 

const; 

virtual const char GctClassName ( ) const; 

class TIteraior ( 

public: 

virtual MCoIleeUble*Ftrst ( ) = 0; 
- 0 virtual MCoUcctible*Next ( ) = 0; 



From these declarations it is clear that MCollectible is a 
base class for common objects and provides an object 
55 comparison and most derived class name access methods. 
TCollection is an abstract base class which defines the 
protocol for collections and it also inherits the protocol of 
MCollectible. We know that execution of TCollection: Add 
requires two objects, an implicit TCollection object which 
60 represents the context of the virtual Add method and an 
MCollectible object to be added to the collection. It is also 
clear that Add may return an MCollectible* as a function 
result We can determine the number of members in a 
TCollection using Count, and we can iterate over members 
65 in a TCollection using the protocol defined by TIterator, an 
abstract base class which defines the protocol for iterating 
over collections. 
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Although these class declarations describe a substantial Invariants 

amount of the protocol to correctly add an MCollectible to j^,^^ ^ predicates which assert target states or 

a TCoUection, addrt.onal senwUcprotocofcare required to characleristics should rcmain constant f or correct target 

correctly add an arbitrary collectible to an arbitrary coUec- Far ex le object invariants determ ine if the 

ton in aU cases. Since semantic protocol cannot be derived s ^ of objects a ^T with ^ ^ ^ ^fed 

directly from a target syntax xt is instead expressed using operation: 0bject invariants ^ fluently osed by the 

objects developed by test engineers. system to ^t^,, tf ^ objects nsed in test sceMrios 

Semantic Protocol arc valid or have been somehow corrupted during the test 

Another type is semantic protocol which describes how 10 acuvlt y- 

the target code should operate from a high leveL operational Invariants are evaluated whenever test targets are affected 

perspective. Semantic protocol describes the high-level directly or indirectly by a test scenario. Test engineers write 

operation of target code which cannot be obtained from a invariants which validate object states rather than write unit 

target syntax alone. tests. Example invariants might assert the following: 

Common sources of abstract semantic protocol are exter- 15 ^ ^ tct Voint&r actually refers to a valid object 

nal requirements specifications, programmer all object references contained within an object are ref- 

documentation, and even source code comments. Program- erences to valid objects 

ruing languages sometimes contain formal constructs for the a system resource exists and is available for use 

representation of a semantic protocol. Since C++ lacks Objects under test can be assigned any number of invari- 

semantic constructs, the subject invention uses objects of 20 ants limited only by system resources. The subject invention 

special classes to represent and evaluate semantic protocol. causes subtype objects to automatically inherit invariants 

Target code semantic protocol is obtained from an analy- assigned to their supertypes. For example, whenever a TSet 

sis of programmer's documentation, external requirements * s checked for invariance, the invariant for generic C++ is 

specifications, and so forth. For the previous example class- first evaluated, then the MCollectible invariant is evaluated, 

esMCollectible and TCollection assume there are two 25 followed by the TCoUcction invariant evaluation, and finally 

semantic rules which must be followed: MColiectibles being the TSet invariant is evaluated. Evaluation of multiple 

compared using IsEqual must have parameter MColiectibles invariants is terminated immediately after the first semantic 

having identical type as this, and non-empty TCollections violation. 

can only contain MColiectibles having identical type. These Data being evaluated for invariance are contained within 

semantic rules disallow IsEqualing a TFoo to a TBar, or 30 data encapsulates descended from TDatum, the abstract 

Adding a TFoo to a TCollection already containing one or base class for target data containers. A TDatum reference is 

more TBars even though both TFoo and TBar are MCol- passed as an explicit argument for invariance from the 

lectibles. Although MCoUecuble::IsEqual and TCollection: encapsulator. Consider the examples in FIG. 3 which illus- 

:Add syntactically specify MCoUectible*parameters, not all (rates a computer listing of an invariant class declaration for 

MColiectibles can be IsEqualcd or Added according to the 35 generic objects, MColiectibles 300 and TCollections 310. 

semantic protocol of the example classes. This example Note the presence of helper methods within each invariant 

illustrates why semantic protocol must be provided to an class which are used to neatly partition the invariant sub 

automated testing system to augment the syntactic protocol checks used by the Test invariance evaluation. FIG. 4 

which by itself is insufficient to describe correct target code illustrates nontrivial invariant method definitions, 

operation. Semantic protocol is represented using predicates 40 Invariants usually consist of a series of checks, and if any 

to evaluate the semantic protocol of target code. check fails, then the invariant returns a result of false 

indicating that the target objects has an incorrect or unex- 
cs pected state. Although these example invariant classes pro- 
Abstractions which represent semantic protocol are called vide limited verification of MColiectibles and TCollection 
predicates. In an abstract sense predicates are expressions of 45 objects, they demonstrate that invariants can be concretely 
operational correctness which can be evaluated to a logical expressed using C++ subroutines. 

result of true or false. A result of true represents semantic Notice the lack of redundancy in FIG. 4 between TCol- 

validity (i.e. something could work or did work), and false lcctiblelnvariant 400 and TCollectionlnvariant even though 

represents semantic invalidity (i.e. something should not a TCollection is a MCollectible. None of the MCollectible 

work or did not work). In the concrete context of the subject 50 checks are performed within the TCollection invariant 

invention, predicate objects perform evaluations to deter- because the system automatically calls the TCollectibleln- 

mine the correctness of the states of an object and the variant on every TCollection before calling TCollectionln- 

operation of subroutines. variant The TCollectionlnvariant can always assume that 

There are two important predicate subtypes: invariants the target object has passed all preceding invariants for 

which validate target states and conditions which validate 55 MCollectible supertypes before being called, 
target actions. Test engineers write invariants for target 

objects and pre and/postconditions far target methods. FIG. ns 

2 illustrates with a block diagram the taxonomy of abstract Conditions are predicates which assert the correctness of 

predicates evaluating semantic protocol. Block 200 contains initial and final conditions for target subroutines explicitly 

the highest level or Predicates. Predicates 200 are either 60 called during test scenarios. There are two types of condi- 

Invariants 220 or Conditions 210. Conditions are either tions: preconditions which assert if subroutines should work. 

Preconditions 230 or Postconditions 240. Engineers must and postconditions which assert if subroutines did work. 

create predicate classes which rejaresent the semantic pro- Both play an important role in driving the automated testing 

to col for the target classes and functions. The predicates are process. Test engineers using the subject invention write pre 

developed by subclassing from TDatalnvariant, TFunction- 65 and postconditions rather than test cases. 

Condition classes and overriding the Test method of the Target subroutines can be assigned a virtually unlimited 

superclasses. number of conditions limited only by system resource 
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constraints. Conditions for virtual methods assigned to maintained and target subroutines operated correctly. Post- 

supertypes are automatically inherited by overriding subtype conditions are primarily responsible for failure detection of 

methods. For example, when asserting conditions for target code and act as inspectors to determine when, where, 

TCoUection: JsEqual, the conditions for M Collectible:: IsE- and how failures arise during automated testing, 

qual are evaluated first. Then, the evaluation of conditions 5 Using the TCoUection::Add(MCoUectible*) example, 

for TCollection:: IsEqual. Evaluation of multiple conditions consider the following concrete postconditions which deter- 

is terrninated immediately after a semantic violation has mine if a coUectible has been correctly added to a collection 

been detected. gi veD ^ fa t precondition asserting that semantic protocol 

~ . . H would not be violated. FIG. 6 at label 610 illustrates a simple 

TOCOnaiaons « postcondition class for TCollection: :Add. 

Preconditions for subroutmes determine if a specific sys- The postcondition desires information concerning the 

tern state and explicit parameter data set which satisfies the precondition state to determine if the Add operated cor- 

subroutine's syntactic protocol also satisfies the subroutine's rectly: how many collectibles were already in the coUection, 

semantic protocol Preconditions are evaluated immediately and was the added collectible already in the collection? 

before potential execution of a target subroutine in auto- 15 Although the subject invention provides a formal system for 

mated test scenarios. If all of the subroutine's preconditions inter-predicate communication, assume that the precondition 

assert that semantic protocol will be maintained, then me data is made available to the postcondition through global 

target subroutine can be executed with the specified data set variables long gPreCount and Boolean gWasMcmbcr. The 

and system state. If any of the preconditions assert that postcondition Test method has the following definition illus- 

semantic protocol will be violated, then the subroutine will 20 trated in FIG. 6 at 610. 

not be executed and the Ration will be recorded for later First &c postcondMon ^ks me existcnce of ^ col . 

analysis. Preconditions act as filters in the testing process to lectiblewithm^^ 

avoid target subroirtme evocations that would violate rollectiUcs in the collection to an ejected value. If the 

semantic protocol and therefore could not be considered M has ^ decreas ^ has increased by 
valid test scenarios 

more than 1, then the postcondition returns false represent- 
Preconditions are represented using subclasses of TFunc- fog a semantic failure probably caused by an implementation 
tionCondition in the same manner that invariants use TData- defect in the target code. As the example illustrates in FIG. 
Invariant. Simple preconditions only need to override a 6, the combination of syntactic and semantic protocol pro- 
single Test method. Preconditions directly access data being vide a useful means to express the behavior of simple classes 
passed to the target subroutine using a TFunctionData object or functions under test. The ability to share information 
which encapsulates all explicit parameters and a function between preconditions and postconditions offers consider- 
result FIG. 5 illustrates an example precondition for TCol- able opportunity to deterrnine if failures have arisen in target 
lection: lAdd at 500. The precondition's method definition code, 
appears at label 510. 

Notice that there is no redundancy between the TCollec- Associates 

tionAddPrecondition and the invariants for MCoUectible . ti . ~ * 

andTCollection. Before calling the precondition, the subject ^ S °^ ™ user-defined rules which explicitly hnk 

invention first calls the invariants using all explicit function F^cates (invariants, preconditions and postconditions) to 

data to assert their validity. The subject invention ensures M ^^f^f^ prions) dedicate 

that all data explicitly passed to a function condition have 40 ° b J e f * m ?* ^oaated with specific 

successfully passed their invariants. This automatic invari- t ?* p *' Assoc f ates ^ w * P^ cate class t0 be 

ant checking simplifies condition development and provides simultaneously associated with multiple targets. For 

consistency of conditions'^ TComparePrecondition to be 

ftettacticmdataaceiOT^ . _ associated w^ boAMCollecu^ 

the target function Add is called. 45 n::IsSame ^ both methods have a semantic protocol 

tp *u a a. *i a * . which requires that the single parameter object being com- 

ff the ftacttoi i date are valid, then the ; F^onditKm pare d nmst have a type identi^l to this, 
performs a semantic check to ensure that adding the MCol- 

lectible to the TCollection maintains the coUection homo- Associates are provided by test engineers in the form of 

genrityusmgmefoUowinglogiciifte 50 statements in declaration scripts. For example, the 

then the coUectible can be added to the collection; if the foUowin g statements represent typical associations, use 

collection is not empty, then the collectible class type must InvariantTMylnvariant from MyTestlib for TBar, TFoo;use 

be identical to the first collectible already in the collection. ^woDdition TaddPrecondition for TCollection: :Add 

If all of these checks yield true, then the coUectible can be (MCoUectible*); 

correctly added to the coUection. 55 Since associates are interpreted using text declarations 

Preconditions can also be used to cache information prior ^ applied & runtime, they provide the flexibility of highly 

to subroutine execution for later postcondition retrieval. For dynamic linking without requiring recompilation of the 

example, when adding a coUectible to a coUection it may be source code - 
necessary to record me number of members in the coUection 

before the coUectible is added, and record if the coUectible 60 Target Sclectors 

was already a member of the coUection. This information Selectors are objects that select targets in automated test 

can be used by the postcondition to determine if the sub- scenarios. There are two primary types of selectors: function 

routine performed correctly. selectors, which select the next function to be tested, and 

Postconditions ^ selectoxs whicn sclcct ^ c parameter data for the cur- 
es rentfy selected function under test. Two default selectors are 

Postconditions are evaluated immediately after execution provided: random selectors which pscudo-randomly select 

of target subroutines to determine if semantic protocol was the ith out of N available targets, and exhaustive selectors 
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which choose all candidate targets at least i times before 
choosing i+1 times. 

Function Wrappers 

Function wrappers are subroutines which are responsible 
for directly executing functions under test Wrappers must 
un encapsulate target function parameter data, execute target 
functions, and reencapsulate the parameter and result data 
from function calls. Although test engineers do not see or 
use wrappers directly, wrappers must be compiled and 
linked into target shared libraries by test engineers. The 
source code for wrappers are generated automatically by the 
subject invention using target source code syntax, and 
should be compiled into target libraries along with predicate 
classes. Wrapper function source code can be created using 
the create wrappers command 

Command Interpreter 

The primary user interface to the subject invention is the 
command interpreter, also known as Q. The command 
interpreter is a text-based, remote controller for develop- 
ment User directives can be entered from CI or indirectly 
via a console command which instructs the subject invention 
to read additional commands contained within a script file. 

There are two categories of directives that the subject 
invention interprets: commands which interactively control 
test execution and declarations which define the properties 
of the test targets. The subject invention currently has two 
independent LeX/Yacc-based parsers for interpreting com- 
mands and declarations which causes a divergence between 
use of the two types of directives: commands can be 
presented directly by using CI or indirectly within script 
files, but declarations are always presented indirectly within 
script files. 

There are two distinct stages which test engineers must 
traverse to test their code using the subject invention. In the 
first stage they must develop predicate classes for their code 
and build shared libraries containing predicate code and 
wrapper functions. In the second stage engineers test their 
target code using scripts or interactively with a human/ 
computer interface. 

There are six major steps in the setup for the subject 
invention testing. 

(1) target code is identified, 

(2) predicate classes are written for target code, 

(3) associates are written which bind predicates to test 
targets, 

(4) the subject invention learns target rules, 

(5) the subject invention creates target wrappers, 

(6) predicate and wrapper sources are compiled, linked, 
and built into a target shared library. 

FIG. 7 represents the major steps in the setup stage. The 
processing commences at function block 700 where the 
designer creates target declarations and inputs them into the 
subject invention. The invention analyzes the target decla- 
rations as depicted in function block 720 and creates target 
wrappers as depicted in function block 722. Then the tester 
creates predicates as shown in function block 730 and inputs 
the predicates and the wrappers into function block 740 
where the tester creates a shared library from the predicates 
and the wrappers. Finally, at function block 750, the target 
shared library is used for testing the application in accor- 
dance with the subject invention. 

Develop Oass Invariants 

Each target class should be accompanied by one or more 
invariants which assert the correctness of a target class 
object's state during automated testing. 
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Function Conditions 

Every public member function in the target code is 
accompanied by one or more preconditions which accurately 
^ and thoroughly assert its semantic pre- and postconditions. 

Predicate Associations 

All invariants and conditions need to be accompanied by 
associations which bind predicates to their intended test 
targets. Associations appear in a declaration script. An 
10 example script appears in the following section. 

Preconditions filter out candidates that are not appropriate 
for a particular design from an automatically generated pool 
of type-legal candidates. Preconditions are concrete func- 
15 tions in C++ which return either true or fals e to represent the 
legality of a candidate subroutine or datum. For example, the 
SetName precondition returns true if a TName candidate 
consists of the characters A-Z, 

FIG. 8 illustrates how a precondition filters out design- 
20 illegal cases from a type-legal test domain. Virtually infinite 
valid test domains can be created by using preconditions to 
filter out invalid scenarios from automatically generated 
type-legal, possibly design-legal domains. This technique 
provides unparalleled efficiency compared to manually writ- 
25 ing test cases. However, target subroutines must have cor- 
responding preconditions which are written by test engi- 
neers. 

Technical writers and client developers benefit from pre- 
condition functions because such functions precisely state 
30 conditions that must be met before a subroutine can be 
called. Tech writer can create concise English instructions 
by disassembling precondition statements. This benefit is 
particularly advantageous when the preconditions state- 
ments in C+f are expressed using high-level expressions. 

35 

Postconditions 

Postconditions determine when failures arise during test 
domain exploration. They are similar to preconditions 
except postconditions are evaluated after each explicit sub- 

40 routine call. Test engineers write postcondition functions 
which verify that each test subroutine call has operated 
according to its specification. For example, the postcondi- 
tion for TPerson::SetName might verify mat the TPerson 
name is identical to the TName instance passed as a param- 

45 eter in the previous SetName call. Like preconditions, all 
target subroutines in the API usually have corresponding 
postconditions. 

Object Testing 

so Object frameworks offer testing challenges such as inher- 
itance correctness, polymorphic substitution, and object 
operations such as construction, destruction, copying, 
assignment, streaming, and comparison over the standard 
procedural testing problems. The subject invention offers 

55 built-in features to automatically verify many of these 
object-oriented features. Preconditions and postconditions 
which are assigned to pure virtual methods in superclasses 
are automatically inherited by concrete subclass methods. 
Supertype invariants are automatically called in order of 

$o precedence when checking the integrity of subtype objects. 
Invariants are called before and after subroutines to check 
correct streaming and cloning for diverse object states which 
would be difficult to reproduce in unit tests. 
The following sections explain how the subject inven- 

65 tion's implementation offers innovative and new techniques 
to resolve several of the reliability and testing issues pre- 
sented earlier. 
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Standard Test Specification 

Since the subject invention algorithmicaUy analyzes data 
type relationships to generate test domains, identical test 
cases for any API will be generated regardless of the test 
engineer involved. Two or more test engineers of differing 
abilities given the same API will generate identical test cases 
for that APL This capability offers a systematic solution to 
fundamental problems for both unit and integration testing. 
In traditional testing, the human effort required to test an API 
has been difficult to predict Often effort was proportional to 
the number of test cases which must be written, but it was 
sometimes impossible to compute the number of cases 
analytically. With the subject invention, the human effort 
required to test is directly proportional to measurable char- 
acteristics of the API under test. 

The human effort required is directly related to the num- 
ber of precondition, postcondition, and invariant functions 
which must be written. Of course, this model assumes that 
all functions require the same effort for all APIs, which is not 
a valid assumption since some APIs are much more complex 
than other. 

However, once a probability distribution of the effort 
(probably in person-minutes) required to write these func- 
tions has been gathered, the total human effort required to 
begin testing an arbitrary API can be estimated using simple 
statistics. 

Automatic Test Case Generation 

As mentioned earlier, the subject invention generates test 
cases automatically when exploring the test domain of an 
APL This implies that no test cases need rewriting after an 
API has been modified. However, the precondition, 
postcondition, and invariant functions for an API might 
require modification. 

Software Reliability Measurement 

Reliability can be automatically measured during testing 
sessions by keeping track of the time spent in explicitly 
called subroutines under test and counting the number of 
failures which have occurred. The mean time between 
failures (MTBF) can be automatically determined because 
both the time under test and the number of failures is 
maintained during automated testing sessions. The software 
reliability can be estimated for any API by statistically 
analysing results of multiple random walk through employ- 
ing the API's design-legal test domain. 

Having test engineers write preconditions, postconditions, 
and invariants is a more efficient strategy than hand-coding 
test cases, particularly as greater numbers of cases are 
needed when testing becomes more rigorous and errors 
become less frequent Testing of object-oriented APIs is 
accommodated by determining type inheritances to allow 
polymorphic candidate date selection, and precondition/ 
postcondition/invariant inheritance for the code under test 
Reuse of objects across sequential messages (method calls) 
offers complex test scenarios which would be virtually 
impossible to create using independent unit tests. 

While the invention has been described in terms of a 
preferred embodiment in a specific system environment 
those skilled in the art recognize that the invention can be 
practiced, with modification, in other and different hardware 
and software environments within the spirit and scope of the 
appended claims. 
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Having thus described our invention, what we claim as 
new, and desire to secure by Letters Patent is: 

1. A method for generating a plurality of valid parameter 
data each of which has an object state and each of which will 

5 operate with a subroutine under test from an application 
under test without causing an error, the method comprising 
the steps of: 

(a) generating a pool of potential parameter data from a 
lexical analysis of the application under test; 

10 (b) storing the pool of potential parameter data in a 
storage; 

(c) examining each potential parameter data of the stored 
pool of potential parameter data and selecting there- 

l5 from data with valid object states as candidate param- 
eter data; and 

(d) examining each candidate parameter data selected in 
step (c) and selecting therefrom valid parameter data 
which will operate with the subroutine without causing 

20 an error condition. 

2. A method far automated testing of a software program 
comprising parameter data and a subroutine operating on the 
parameter data to generate output data, the method, com- 
prising the steps of: 

25 (a) examining the parameter data and generating a data 
consistent signal when the parameter data is internally 
consistent; 

(b) examining the parameter data and generating a data 
valid signal when the parameter data will operate with 

30 the subroutine without causing an error condition; 

(c) running the subroutine with the parameter data when 
the data consistent signal and the data valid signal are 
present; and 

(d) examining the parameter data and output data gener- 
35 ated by the subroutine from the parameter data to 

determine whether an error occurred during operation 
of the subroutine. 

3. A method for automated testing of a software program 
according to claim 2, further comprising the steps of: 

40 (a) receiving user commands; and 

(b) generating wrapper code for running the subroutine 
with the parameter data. 

4. A method for automated testing of a software program 
45 according to claim 2, wherein step (b) includes the step of 

discontinuing automated testing when the data valid signal 
indicates that the parameter data is not valid 

5. A method fox automated testing of a software program 
according to claim 2, further comprising the step of exam- 

5Q ining the parameter data after the subroutine has been run 
and determining whether the parameter data is still internally 
consistent 

6. A method for automated testing of a software program 
according to claim 2, wherein the subroutine requires param- 

55 eter data with a plurality of predetermined data types and 
wherein step (a) comprises the steps of: 
(al) generating a plurality of test sets of parameter data, 
each of the plurality of test sets of parameter data 
having data with one of the plurality of predetermined 
data types; and 
(a2) selecting the test sets of parameter data wherein the 
parameter data is internally consistent 

7. A method far automated testing of a software program 
according to claim 2, wherein step al, further comprises the 

65 steps of: 

a3 . providing an invariant rule for each of the plurality of 
predetermined data types; and 
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a4. associating aa invariant rule with cadi of the plurality 
of test sets of parameter data. 

8. A method for automated testing of a software program 
according to claim 2, wherein step (b) further comprises the 
step of examining the test sets of parameter data selected in 
step (a2) and further selecting test sets of parameter data 
wherein the parameter data will operate with the subroutine 
without causing an error condition. 

9. A method for automated testing of a software program 
according to claim 2, wherein step (a) further comprises the 
steps of: 

(a3) providing an invariant rule for each of the plurality of 

predetermined data types; and 
(a4) associating an invariant rule with each of the plurality 

of test sets of parameter data. 

10. A method for generating a plurality of valid parameter 
data objects each of which has an object state and each of 
which will operate with a subroutine under test without 
causing an error, comprising the steps of: 

(a) generating a pool of potential parameter data objects; 

(b) storing the pool of potential parameter data objects; 

(c) examining each potential parameter data object in the 
stored pool of potential parameter data objects and 
selecting candidate parameter data objects with valid 
object states; and 

(d) examining each candidate parameter data object 
selected in step (c) and further selecting valid param- 
eter data objects which will operate with the subroutine 
without causing an error condition. 
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11. A method for generating a plurality of valid parameter 
data objects according to claim 10. wherein each of the 
potential parameter data objects has a predetermined data 
type and step c, comprises the steps of: 

5 (cl) providing an invariant rule for each of the plurality of 
predetermined data types; and 

(c2) associating an invariant rule with each of the poten- 
tial parameter data objects. 

12. A method for testing whether a subroutine, operating 
10 on parameter data with internal data states to generate output 

data, meets predetermined design requirements, the method 
comprising the steps of: 

(a) storing the subroutine and the parameter data in a 
15 storage; 

(b) examining each piece of parameter data and selecting 
therefrom parameter data having valid internal data 
states as candidate parameter data pieces, which have a 
data syntax which will operate with the subroutine; 

20 (c) examining the candidate parameter data selected in 
step (b) and selecting therefrom valid parameter data 
having data semantics in accordance with the prede- 
termined design requirements; 

(d) running the subroutine with the valid parameter data 
25 selected in step (c); and 

(e) examining output data generated by the subroutine as 
a result of step (d) to determine whether the subroutine 
met design requirements. 

***** 
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